# Load the dataset
import pandas as pd
df=pd.read_csv("WA_Fn-UseC_-HR-Employee-Attrition.csv")

# Display the first few rows of the dataset in a visually appealing format
print("Dataset Loaded Successfully!")
display(df.head())

Dataset Loaded Successfully!

# Display basic information about the dataset
print("\nDataset Information:")
df.info()

Dataset Information:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1470 entries, 0 to 1469
Data columns (total 35 columns):
 #   Column                    Non-Null Count  Dtype 
---  ------                    --------------  ----- 
 0   Age                       1470 non-null   int64 
 1   Attrition                 1470 non-null   object
 2   BusinessTravel            1470 non-null   object
 3   DailyRate                 1470 non-null   int64 
 4   Department                1470 non-null   object
 5   DistanceFromHome          1470 non-null   int64 
 6   Education                 1470 non-null   int64 
 7   EducationField            1470 non-null   object
 8   EmployeeCount             1470 non-null   int64 
 9   EmployeeNumber            1470 non-null   int64 
 10  EnvironmentSatisfaction   1470 non-null   int64 
 11  Gender                    1470 non-null   object
 12  HourlyRate                1470 non-null   int64 
 13  JobInvolvement            1470 non-null   int64 
 14  JobLevel                  1470 non-null   int64 
 15  JobRole                   1470 non-null   object
 16  JobSatisfaction           1470 non-null   int64 
 17  MaritalStatus             1470 non-null   object
 18  MonthlyIncome             1470 non-null   int64 
 19  MonthlyRate               1470 non-null   int64 
 20  NumCompaniesWorked        1470 non-null   int64 
 21  Over18                    1470 non-null   object
 22  OverTime                  1470 non-null   object
 23  PercentSalaryHike         1470 non-null   int64 
 24  PerformanceRating         1470 non-null   int64 
 25  RelationshipSatisfaction  1470 non-null   int64 
 26  StandardHours             1470 non-null   int64 
 27  StockOptionLevel          1470 non-null   int64 
 28  TotalWorkingYears         1470 non-null   int64 
 29  TrainingTimesLastYear     1470 non-null   int64 
 30  WorkLifeBalance           1470 non-null   int64 
 31  YearsAtCompany            1470 non-null   int64 
 32  YearsInCurrentRole        1470 non-null   int64 
 33  YearsSinceLastPromotion   1470 non-null   int64 
 34  YearsWithCurrManager      1470 non-null   int64 
dtypes: int64(26), object(9)
memory usage: 402.1+ KB

# Check for missing values
missing_values = df.isnull().sum()

# Display missing values count for each column
print("Missing Values in Each Column:")
print(missing_values)

Missing Values in Each Column:
Age                         0
Attrition                   0
BusinessTravel              0
DailyRate                   0
Department                  0
DistanceFromHome            0
Education                   0
EducationField              0
EmployeeCount               0
EmployeeNumber              0
EnvironmentSatisfaction     0
Gender                      0
HourlyRate                  0
JobInvolvement              0
JobLevel                    0
JobRole                     0
JobSatisfaction             0
MaritalStatus               0
MonthlyIncome               0
MonthlyRate                 0
NumCompaniesWorked          0
Over18                      0
OverTime                    0
PercentSalaryHike           0
PerformanceRating           0
RelationshipSatisfaction    0
StandardHours               0
StockOptionLevel            0
TotalWorkingYears           0
TrainingTimesLastYear       0
WorkLifeBalance             0
YearsAtCompany              0
YearsInCurrentRole          0
YearsSinceLastPromotion     0
YearsWithCurrManager        0
dtype: int64

# Check for duplicates in the dataset
duplicates = df.duplicated()

# Display the number of duplicate rows
num_duplicates = duplicates.sum()
print(f'Number of duplicate rows: {num_duplicates}')

# If duplicates exist, we can drop them
if num_duplicates > 0:
    # Drop duplicate rows
    df = df.drop_duplicates()
    print("Duplicates have been removed from the dataset.")
else:
    print("No duplicates found in the dataset.")

Number of duplicate rows: 0
No duplicates found in the dataset.

# List of ordinal columns
from sklearn.preprocessing import LabelEncoder
ordinal_columns = ['EnvironmentSatisfaction', 'JobInvolvement', 'JobSatisfaction',
                   'PerformanceRating', 'RelationshipSatisfaction', 'WorkLifeBalance']

# Initialize the LabelEncoder
label_encoder = LabelEncoder()

# Apply Label Encoding for each ordinal column
for col in ordinal_columns:
    df[col] = label_encoder.fit_transform(df[col])

# Display a sample of the DataFrame after label encoding
df[ordinal_columns].head()

import pandas as pd
# List of nominal columns (categorical columns)
nominal_columns = ['BusinessTravel', 'Department', 'EducationField', 'Gender', 
                   'JobRole', 'MaritalStatus', 'OverTime', 'StockOptionLevel']

# Apply One-Hot Encoding for nominal columns
df_encoded = pd.get_dummies(df, columns=nominal_columns, drop_first=True)

# Display the updated DataFrame with one-hot encoded columns
df_encoded.head()

from sklearn.preprocessing import StandardScaler

# List of numerical columns to be standardized
numerical_columns = ['MonthlyIncome', 'MonthlyRate']

# Initialize StandardScaler (Z-score normalization)
scaler = StandardScaler()

# Apply the scaler to the numerical columns
df[numerical_columns] = scaler.fit_transform(df[numerical_columns])

# Show the first few rows of the transformed numerical columns
print("\nFirst few rows of the scaled numerical columns:")
print(df[numerical_columns].head())

# Check the shape of the dataset after processing
print("\nShape of the dataset after processing:", df.shape)

First few rows of the scaled numerical columns:
   MonthlyIncome  MonthlyRate
0      -0.108350     0.726020
1      -0.291719     1.488876
2      -0.937654    -1.674841
3      -0.763634     1.243211
4      -0.644858     0.325900

Shape of the dataset after processing: (1470, 35)

# Drop irrelevant columns from the dataset
df_cleaned = df.drop(columns=['EmployeeCount', 'StandardHours', 'PerformanceRating','EmployeeNumber'])

# Check the shape of the dataset after dropping columns
print(df_cleaned.shape)

(1470, 31)

df_cleaned.describe().T

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd


# List of numerical columns to perform univariate analysis on
numerical_cols = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany'
]

# Number of columns for subplots (adjust based on your preference)
n_cols = 3

# Number of rows based on the number of columns
n_rows = len(numerical_cols) // n_cols + (len(numerical_cols) % n_cols > 0)

# Setting up the plotting area
fig, axes = plt.subplots(n_rows, n_cols, figsize=(18, 5 * n_rows))  # Adjust height as needed
fig.subplots_adjust(hspace=0.5, wspace=0.3)  # Adjust the space between subplots

# Loop through and create histogram for each column
for i, col in enumerate(numerical_cols):
    row = i // n_cols  # Get the row index
    col_index = i % n_cols  # Get the column index
    
    sns.histplot(df[col], bins=30, kde=False, ax=axes[row, col_index])
    axes[row, col_index].set_title(f'Histogram of {col}')
    axes[row, col_index].set_xlabel(col)
    axes[row, col_index].set_ylabel('Frequency')

# Remove empty subplots if there are any
for j in range(i + 1, n_rows * n_cols):
    fig.delaxes(axes[j // n_cols, j % n_cols])

# Display the plots
plt.show()

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Assuming the dataset is loaded into 'df'
# List of numerical columns to perform univariate analysis on
numerical_cols = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany'
]

# Number of columns for subplots (adjust based on your preference)
n_cols = 4  # You can change this value for a different number of columns per row

# Number of rows based on the number of columns
n_rows = len(numerical_cols) // n_cols + (len(numerical_cols) % n_cols > 0)

# Setting up the plotting area for box plots
fig, axes = plt.subplots(n_rows, n_cols, figsize=(18, 5 * n_rows))  # Adjust height as needed
fig.subplots_adjust(hspace=0.5, wspace=0.3)  # Adjust the space between subplots

# Loop through and create boxplot for each column
for i, col in enumerate(numerical_cols):
    row = i // n_cols  # Get the row index
    col_index = i % n_cols  # Get the column index
    
    # Create Boxplot
    sns.boxplot(x=df[col], ax=axes[row, col_index], color='orange')
    axes[row, col_index].set_title(f'Boxplot of {col}')
    axes[row, col_index].set_xlabel(col)

# Remove empty subplots if there are any
for j in range(i + 1, n_rows * n_cols):
    fig.delaxes(axes[j // n_cols, j % n_cols])

# Display the plots
plt.show()

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Assuming the dataset is loaded into 'df'
# List of numerical columns to perform univariate analysis on
numerical_cols = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany'
]

# Number of columns for subplots (adjust based on your preference)
n_cols = 4  # You can change this value for a different number of columns per row

# Number of rows based on the number of columns
n_rows = len(numerical_cols) // n_cols + (len(numerical_cols) % n_cols > 0)

# Setting up the plotting area for density plots
fig, axes = plt.subplots(n_rows, n_cols, figsize=(18, 5 * n_rows))  # Adjust height as needed
fig.subplots_adjust(hspace=0.5, wspace=0.3)  # Adjust the space between subplots

# Loop through and create density plot for each column
for i, col in enumerate(numerical_cols):
    row = i // n_cols  # Get the row index
    col_index = i % n_cols  # Get the column index
    
    # Create Density Plot
    sns.kdeplot(df[col], ax=axes[row, col_index], color='green')
    axes[row, col_index].set_title(f'Density Plot of {col}')
    axes[row, col_index].set_xlabel(col)
    axes[row, col_index].set_ylabel('Density')

# Remove empty subplots if there are any
for j in range(i + 1, n_rows * n_cols):
    fig.delaxes(axes[j // n_cols, j % n_cols])

# Display the plots
plt.show()

import itertools

# List of numerical columns
numerical_cols = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany'
]

# Correlation matrix heatmap
plt.figure(figsize=(12, 8))
sns.heatmap(df[numerical_cols].corr(), annot=True, fmt='.2f', cmap='coolwarm', cbar=True)
plt.title('Correlation Matrix for All Numerical Variables')
plt.show()

from itertools import combinations
import math

# List of numerical columns
numerical_cols = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany'
]

# Generate all combinations of numerical columns
combinations_list = list(combinations(numerical_cols, 2))

# Calculate rows and columns dynamically
n_plots = len(combinations_list)
n_cols = 4  # Number of columns in the grid
n_rows = math.ceil(n_plots / n_cols)  # Calculate the required number of rows

# Plot scatter plots for all pairs
plt.figure(figsize=(n_cols * 5, n_rows * 4))  # Adjust size dynamically
for i, (x, y) in enumerate(combinations_list, 1):
    plt.subplot(n_rows, n_cols, i)
    sns.scatterplot(data=df, x=x, y=y, hue='Attrition', alpha=0.6)
    plt.title(f'{x} vs {y}')
    plt.xlabel(x)
    plt.ylabel(y)

plt.tight_layout()
plt.show()

import seaborn as sns
import matplotlib.pyplot as plt

# Select numerical columns for the pair plot
numerical_cols = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany'
]

# Create a pair plot with appropriate aesthetics
sns.pairplot(
    data=df,
    vars=numerical_cols,
    hue='Attrition',  # Use 'Churn Value' to differentiate
    palette='coolwarm',  # Use a visually clear color palette
    diag_kind='kde',  # Show KDE on the diagonal
    markers=["o", "s"],  # Distinguish points by marker
    height=2.5  # Adjust the size of each plot
)

# Display the plot
plt.show()

# Import necessary libraries
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Select relevant numerical columns for multivariate analysis
numerical_cols = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany'
]

# 1. Heatmap
plt.figure(figsize=(10, 8))
correlation_matrix = df[numerical_cols].corr()
sns.heatmap(correlation_matrix, annot=True, fmt=".2f", cmap="coolwarm", cbar=True)
plt.title("Correlation Heatmap")
plt.show()

import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Specify the columns to analyze
columns_to_analyze = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome',
    'TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance', 'YearsAtCompany', 'Attrition'
]

# Filter the DataFrame for the selected columns
analysis_df = df[columns_to_analyze]

### 2. Pair Plot
sns.pairplot(analysis_df, hue='Attrition', palette='husl', diag_kind='kde')

plt.suptitle("Pairwise Relationships", y=1.02)
plt.show()

### 3. Principal Component Analysis (PCA)
# Standardize the data
scaler = StandardScaler()
scaled_data = scaler.fit_transform(analysis_df.drop(columns=['Attrition']))

# Apply PCA
pca = PCA(n_components=2)
pca_transformed = pca.fit_transform(scaled_data)

# Create a DataFrame for the PCA results
pca_df = pd.DataFrame(data=pca_transformed, columns=['PC1', 'PC2'])
pca_df['Attrition'] = analysis_df['Attrition'].reset_index(drop=True)

# Scatter Plot of PCA Results
plt.figure(figsize=(8, 6))
sns.scatterplot(data=pca_df, x='PC1', y='PC2', hue='Attrition', palette='Set2', alpha=0.7)
plt.title("PCA Scatter Plot")
plt.xlabel("Principal Component 1")
plt.ylabel("Principal Component 2")
plt.legend(title="Attrition")
plt.show()

### 4. Insights
# Explained variance for each component
explained_variance = pca.explained_variance_ratio_
print(f"Explained Variance by PCA Components: {explained_variance}")

Explained Variance by PCA Components: [0.30759207 0.11913809]

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LinearRegression, LogisticRegression, Lasso, Ridge, ElasticNet
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score, accuracy_score, precision_score, recall_score, f1_score, roc_auc_score

df. columns

Index(['Age', 'Attrition', 'BusinessTravel', 'DailyRate', 'Department',
       'DistanceFromHome', 'Education', 'EducationField', 'EmployeeCount',
       'EmployeeNumber', 'EnvironmentSatisfaction', 'Gender', 'HourlyRate',
       'JobInvolvement', 'JobLevel', 'JobRole', 'JobSatisfaction',
       'MaritalStatus', 'MonthlyIncome', 'MonthlyRate', 'NumCompaniesWorked',
       'Over18', 'OverTime', 'PercentSalaryHike', 'PerformanceRating',
       'RelationshipSatisfaction', 'StandardHours', 'StockOptionLevel',
       'TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
       'YearsAtCompany', 'YearsInCurrentRole', 'YearsSinceLastPromotion',
       'YearsWithCurrManager'],
      dtype='object')

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

# Load your dataset (assuming 'df' is your DataFrame)

# Encode the 'Attrition' column
le = LabelEncoder()
df['Attrition_Encoded'] = le.fit_transform(df['Attrition'])

X = df[['TotalWorkingYears']]
y = df['Attrition_Encoded']  # Use the encoded version of Attrition

# Split the data into training and testing sets (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Perform Simple Linear Regression
model = LinearRegression()
model.fit(X_train, y_train)

# Predict on test data
y_pred = model.predict(X_test)

# Evaluate the model performance
r2 = r2_score(y_test, y_pred)
print(f'R²: {r2}')

rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(f'RMSE: {rmse}')

# Visualize the regression line and the residuals
plt.figure(figsize=(10, 6))

# Plotting the regression line
plt.subplot(1, 2, 1)
plt.scatter(X_test, y_test, color='blue', label='Actual data')
plt.plot(X_test, y_pred, color='red', label='Regression Line')
plt.title('Simple Linear Regression')
plt.xlabel('Total Working Years')
plt.ylabel('Attrition (Encoded)')
plt.legend()

# Plotting residuals
residuals = y_test - y_pred
plt.subplot(1, 2, 2)
plt.scatter(X_test, residuals, color='green')
plt.axhline(0, color='black', linewidth=1)
plt.title('Residuals')
plt.xlabel('Total Working Years')
plt.ylabel('Residuals')

plt.tight_layout()
plt.show()

R²: 0.021801909470947067
RMSE: 0.335481416976357

# Import necessary libraries
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import statsmodels.api as sm

# Load your dataset (assuming 'df' is your DataFrame)
# Define independent variables (numerical predictors)
columns_to_analyze = [
    'Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome',
    'TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance', 'YearsAtCompany', 'Attrition'
]

# Define dependent variable (target variable, here we use 'Churn Value' or 'Churn Label')
y = df['Attrition']  # or use df['Churn Label']

# Define independent variables (X)
X = df[numerical_cols]

# Add constant term to X for intercept in statsmodels
X = sm.add_constant(X)

# Split the data into training and testing sets (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 1. Perform Multiple Linear Regression using sklearn
model = LinearRegression()
model.fit(X_train, y_train)

# 2. Evaluate the model performance on the test data
y_pred = model.predict(X_test)

# R² (R-squared)
r2 = r2_score(y_test, y_pred)
print(f'R²: {r2}')

# RMSE (Root Mean Squared Error)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(f'RMSE: {rmse}')

# 3. Significance of each predictor variable using statsmodels
# Train the model using statsmodels (for p-values and coefficient analysis)
X_train_sm = sm.add_constant(X_train)  # Add constant for intercept
sm_model = sm.OLS(y_train, X_train_sm).fit()

# Print the summary to see p-values and coefficients
print(sm_model.summary())

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np
import matplotlib.pyplot as plt

# Step 1: Prepare the Data
X = df[['TotalWorkingYears']]  # Keep it as a DataFrame
y = df['Attrition'].values

# Step 2: Apply Polynomial Features (degree = 3, you can change degree based on your preference)
poly = PolynomialFeatures(degree=3)
X_poly = poly.fit_transform(X)

# Step 3: Fit the Polynomial Regression Model
poly_model = LinearRegression()
poly_model.fit(X_poly, y)

# Step 4: Make Predictions
y_poly_pred = poly_model.predict(X_poly)

# Step 5: Evaluate the Polynomial Regression Model
r2_poly = r2_score(y, y_poly_pred)
rmse_poly = np.sqrt(mean_squared_error(y, y_poly_pred))
mae_poly = mean_absolute_error(y, y_poly_pred)

print(f"Polynomial Regression - R²: {r2_poly}")
print(f"Polynomial Regression - RMSE: {rmse_poly}")
print(f"Polynomial Regression - MAE: {mae_poly}")

# Visualize the simple linear regression line
plt.scatter(X, y, color='blue', label='Actual values')

# Predict using simple linear regression (just for comparison)
linear_model = LinearRegression()
linear_model.fit(X, y)
y_linear_pred = linear_model.predict(X)
plt.plot(X, y_linear_pred, color='red', label='Linear Regression', linewidth=2)

# Visualize the polynomial regression curve
X_poly_sorted = np.sort(X.values, axis=0)  # Sorting for better curve plotting
y_poly_sorted = poly_model.predict(poly.transform(X_poly_sorted))

plt.plot(X_poly_sorted, y_poly_sorted, color='green', label='Polynomial Regression', linewidth=2)
plt.title('Polynomial vs Linear Regression')
plt.xlabel('Monthly Charges')
plt.ylabel('Churn Value')
plt.legend()
plt.show()

# Import necessary libraries
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, roc_auc_score, confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns

# Assuming 'df' is the DataFrame and 'Churn Value' is the target variable
# Prepare the features (X) and target (y)
X = df.drop(columns=['Attrition'])
y = df['Attrition']

# Convert categorical variables to numerical (one-hot encoding)
X = pd.get_dummies(X, drop_first=True)

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Standardize the features for logistic regression
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Initialize and fit the logistic regression model
log_reg = LogisticRegression(max_iter=1000)
log_reg.fit(X_train, y_train)

# Make predictions
y_pred = log_reg.predict(X_test)

# Calculate accuracy, precision, recall, and ROC-AUC
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, log_reg.predict_proba(X_test)[:, 1])

# Print the evaluation metrics
print(f'Accuracy: {accuracy:.4f}')
print(f'Precision: {precision:.4f}')
print(f'Recall: {recall:.4f}')
print(f'ROC-AUC: {roc_auc:.4f}')

# Confusion Matrix
conf_matrix = confusion_matrix(y_test, y_pred)
plt.figure(figsize=(6, 5))
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Greens', cbar=False)
plt.title('Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.show()

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Lasso, Ridge, ElasticNet
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np

# Define the features and target variable
X = df[['Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome',
    'TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance', 'YearsAtCompany', 'Attrition']]  # Example features
y = df['Attrition']  # Target variable

# Split the data into train and test sets (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize the features (important for regularization models)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Initialize the models with regularization parameters
lasso = Lasso(alpha=0.1)  # Lasso with L1 penalty
ridge = Ridge(alpha=0.1)  # Ridge with L2 penalty
elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)  # ElasticNet with both L1 and L2 penalties

# Train the models
lasso.fit(X_train_scaled, y_train)
ridge.fit(X_train_scaled, y_train)
elastic_net.fit(X_train_scaled, y_train)

# Make predictions on the test set
y_pred_lasso = lasso.predict(X_test_scaled)
y_pred_ridge = ridge.predict(X_test_scaled)
y_pred_elastic_net = elastic_net.predict(X_test_scaled)

# Evaluate the models using Regression Metrics: R², RMSE, and MAE
def evaluate_model(y_true, y_pred):
    r2 = r2_score(y_true, y_pred)
    rmse = np.sqrt(mean_squared_error(y_true, y_pred))  # RMSE (Root Mean Squared Error)
    mae = mean_absolute_error(y_true, y_pred)  # MAE (Mean Absolute Error)
    return r2, rmse, mae

# Evaluate each model
lasso_r2, lasso_rmse, lasso_mae = evaluate_model(y_test, y_pred_lasso)
ridge_r2, ridge_rmse, ridge_mae = evaluate_model(y_test, y_pred_ridge)
elastic_net_r2, elastic_net_rmse, elastic_net_mae = evaluate_model(y_test, y_pred_elastic_net)

# Display the results
print("LASSO Regression - R²:", lasso_r2, "RMSE:", lasso_rmse, "MAE:", lasso_mae)
print("Ridge Regression - R²:", ridge_r2, "RMSE:", ridge_rmse, "MAE:", ridge_mae)
print("Elastic Net Regression - R²:", elastic_net_r2, "RMSE:", elastic_net_rmse, "MAE:", elastic_net_mae)

from statsmodels.regression.quantile_regression import QuantReg
import numpy as np

# Assume you have X_train and y_train
# Fit the quantile regression model for the median (quantile = 0.5)
model = QuantReg(y_train, X_train)
quantile_model = model.fit(q=0.5)

# Print summary
print(quantile_model.summary())

# Predict quantile (for example, median)
predictions = quantile_model.predict(X_test)

import statsmodels.api as sm
import pandas as pd

# Example with count data
# Assume X_train and y_train
model = sm.GLM(y_train, X_train, family=sm.families.Poisson()).fit()

# Print summary
print(model.summary())

# Predict values
predictions = model.predict(X_test)

from statsmodels.genmod.generalized_linear_model import GLM
from statsmodels.genmod.families import Poisson
import pandas as pd

# Negative binomial regression assumes overdispersion
model = GLM(y_train, X_train, family=sm.families.NegativeBinomial()).fit()

# Print summary
print(model.summary())

# Predict values
predictions = model.predict(X_test)

from statsmodels.discrete.count_model import ZeroInflatedPoisson
import pandas as pd

# Fit zero-inflated Poisson regression model
model = ZeroInflatedPoisson(y_train, X_train).fit()

# Print summary
print(model.summary())

# Predict values
predictions = model.predict(X_test)

from sklearn.linear_model import LogisticRegression
from statsmodels.genmod.families import Poisson

# Logistic regression for zero vs non-zero
logit_model = LogisticRegression().fit(X_train, y_train == 0)

# Fit count regression on positive counts
count_model = sm.GLM(y_train[y_train > 0], X_train[y_train > 0], family=Poisson()).fit()

!pip install lifelines
from lifelines import CoxPHFitter
import pandas as pd

# Sample data for demonstration
# Define the duration, event, and covariates
duration = [5, 6, 6, 2, 4]  # Example durations
event = [1, 0, 1, 1, 0]      # Example event indicators (1 = event occurred, 0 = censored)
X1 = [10, 20, 10, 30, 20]    # Example covariate 1
X2 = [1, 2, 1, 2, 1]          # Example covariate 2

# Create a DataFrame with the defined variables
data = pd.DataFrame({'duration': duration, 'event': event, 'X1': X1, 'X2': X2})

# Fit Cox Proportional Hazards model
cph = CoxPHFitter()
cph.fit(data, duration_col='duration', event_col='event')

# Print summary
cph.print_summary()

# Predict hazard ratio or survival function
# Assuming X_test is defined similarly to the covariates in the data
X_test = pd.DataFrame({'X1': [15], 'X2': [1]})  # Example test data
predictions = cph.predict_partial_hazard(X_test)

from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline


# Define the features and target variable
X = df[['Age', 'DistanceFromHome', 'Education', 'JobSatisfaction', 'MonthlyIncome','TotalWorkingYears', 'TrainingTimesLastYear', 'WorkLifeBalance',
    'YearsAtCompany']]  
y = df['Attrition']  # Target variable

# Split the data into train and test sets (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

pca = PCA(n_components=5)
reg = LinearRegression()

# Create a pipeline combining PCA and linear regression
pcr_model = make_pipeline(pca, reg)
pcr_model.fit(X_train, y_train)

# Predict values
predictions = pcr_model.predict(X_test)

print(predictions)

from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

# Assume X_train, y_train is your dataset
pca = PCA(n_components=5)
reg = LinearRegression()

# Create a pipeline combining PCA and linear regression
pcr_model = make_pipeline(pca, reg)
pcr_model.fit(X_train, y_train)

# Predict values
predictions = pcr_model.predict(X_test)
print(predictions)

logit = LogisticRegression(max_iter=1000)

df.head(1)

df['TotalWorkingYears'].plot(kind='hist')

from scipy import stats


# Perform the Kolmogorov-Smirnov test
d_statistic, pval = stats.kstest(df['TotalWorkingYears'], 'norm', args=(df['TotalWorkingYears'].mean(), df['TotalWorkingYears'].std()))

if pval < 0.05:
    print('not normal')
else:
    print('normal')

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer  # Importing SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder  # Importing necessary preprocessing classes
from sklearn.decomposition import PCA  # Importing PCA
from sklearn.compose import ColumnTransformer  # Importing ColumnTransformer
from sklearn.feature_selection import RFE  # Importing RFE
from imblearn.over_sampling import SMOTE  # Importing SMOTE for resampling
import numpy as np  # Importing numpy for handling NaN values

# Define SMOTE instance
smote = SMOTE()  # Create an instance of SMOTE


# Define one_hot_cols as needed for your categorical columns
one_hot_cols = []  # Replace with your actual categorical column names

# for monthly and total charges
logit_pipe_num = Pipeline([
    ('imputer', SimpleImputer(strategy='median', missing_values=np.nan)),
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=1))
])

# for all object columns
logit_pipe_cat = Pipeline([
    ('onehot', OneHotEncoder(drop='first')),
])

# transforming all columns
logit_transformer = ColumnTransformer([
    ('pipe_num', logit_pipe_num, ['MonthlyIncome', 'TotalWorkingYears']),
    ('pipe_cat', logit_pipe_cat, one_hot_cols)
])

# combine all pipeline
logit_pipe_combine = Pipeline([
    ('transformer', logit_transformer),
    ('rfe', RFE(logit)),
    ('resampling', smote),
    ('logit', logit)
])

from sklearn.metrics import get_scorer_names

# Get the available scorer names
scorers = get_scorer_names()

# Print the list of scorers
print(scorers)

from imblearn.over_sampling import SMOTE
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score
from imblearn.pipeline import Pipeline as ImbPipeline  # Corrected import

# Define your pipeline using ImbPipeline
logit_pipe_combine = ImbPipeline(steps=[
    ('scaler', StandardScaler()),  # Scaling step
    ('smote', SMOTE(random_state=2021)),  # SMOTE step for oversampling
    ('logit', LogisticRegression())  # Logistic Regression step
])

# Define your RepeatedStratifiedKFold
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=2021)

# Cross-validation with recall as the scoring metric
logit_score = cross_val_score(logit_pipe_combine, X_train, y_train, scoring='recall', cv=rskf, n_jobs=-1, verbose=1)

# Print the scores
print(logit_score)

print('Logit Val Score:', logit_score)

logit_pipe_combine.get_params()

from sklearn.model_selection import GridSearchCV
# Define the hyperparameters grid to tune
param_grid = {
    'logit__C': [0.1, 1, 10],  # Regularization strength
    'logit__solver': ['liblinear', 'saga']  # Solvers for logistic regression
}
# Perform grid search with cross-validation
logit_grid = GridSearchCV(logit_pipe_combine, param_grid, scoring='recall', cv=5, n_jobs=-1, verbose=1)
# Fit the grid search on the training data
logit_grid.fit(X_train, y_train)
logit_tuned = logit_grid.best_estimator_
logit_tuned_score = cross_val_score(logit_tuned, X_train, y_train, scoring='recall', cv=rskf, n_jobs=-1, verbose=1)
logit_tuned_score

plt.plot(np.arange(len(logit_tuned_score)), logit_tuned_score)
plt.ylim(0,1)
plt.show()

logit_tuned_score.mean() # tuned dari cross val score

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Fit the tuned model
logit_tuned.fit(X_train, y_train)

# Predict probabilities for the positive class
y_probs = logit_tuned.predict_proba(X_test)[:, 1]

# Compute precision-recall curve
precision, recall, _ = precision_recall_curve(y_test, y_probs)

# Plot precision-recall curve
plt.figure(figsize=(6, 5))
plt.plot(recall, precision, color='b', label='Precision-Recall curve')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.legend(loc='best')
plt.show()

proba1 = logit_tuned.predict_proba(X_test)[:,1]
y_pred = logit_tuned.predict(X_test)
thresh =0.341275
pred_03 = np.where(proba1 > thresh, 1, 0)
res_df = pd.DataFrame({'proba1': proba1, 'y_pred': y_pred, 'y_pred03': pred_03})
plt.figure(figsize=(16,8))
plt.subplot(121)
sns.scatterplot(x=range(len(res_df)), y=res_df['proba1'], hue=res_df['y_pred03'])
plt.axhline(thresh, color='red')

plt.subplot(122)
sns.scatterplot(x=range(len(res_df)), y=res_df['proba1'], hue=res_df['y_pred'])
plt.axhline(0.5, color='red')

# Import necessary libraries
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, roc_auc_score, confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns

# Assuming 'df' is the DataFrame and 'Churn Value' is the target variable
# Prepare the features (X) and target (y)
X = df.drop(columns=['Attrition'])
y = df['Attrition']

# Convert categorical variables to numerical (one-hot encoding)
X = pd.get_dummies(X, drop_first=True)

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Standardize the features for logistic regression
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Initialize and fit the logistic regression model
log_reg = LogisticRegression(max_iter=1000)
log_reg.fit(X_train, y_train)

# Make predictions
y_pred = log_reg.predict(X_test)

# Calculate accuracy, precision, recall, and ROC-AUC
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, log_reg.predict_proba(X_test)[:, 1])

# Print the evaluation metrics
print(f'Accuracy: {accuracy:.4f}')
print(f'Precision: {precision:.4f}')
print(f'Recall: {recall:.4f}')
print(f'ROC-AUC: {roc_auc:.4f}')

# Confusion Matrix
conf_matrix = confusion_matrix(y_test, y_pred)
plt.figure(figsize=(6, 5))
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Greens', cbar=False)
plt.title('Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.show()

	count	mean	std	min	25%	50%	75%	max
Age	1470.0	3.692381e+01	9.135373	18.000000	30.000000	36.000000	43.000000	60.000000
DailyRate	1470.0	8.024857e+02	403.509100	102.000000	465.000000	802.000000	1157.000000	1499.000000
DistanceFromHome	1470.0	9.192517e+00	8.106864	1.000000	2.000000	7.000000	14.000000	29.000000
Education	1470.0	2.912925e+00	1.024165	1.000000	2.000000	3.000000	4.000000	5.000000
EnvironmentSatisfaction	1470.0	1.721769e+00	1.093082	0.000000	1.000000	2.000000	3.000000	3.000000
HourlyRate	1470.0	6.589116e+01	20.329428	30.000000	48.000000	66.000000	83.750000	100.000000
JobInvolvement	1470.0	1.729932e+00	0.711561	0.000000	1.000000	2.000000	2.000000	3.000000
JobLevel	1470.0	2.063946e+00	1.106940	1.000000	1.000000	2.000000	3.000000	5.000000
JobSatisfaction	1470.0	1.728571e+00	1.102846	0.000000	1.000000	2.000000	3.000000	3.000000
MonthlyIncome	1470.0	-4.471102e-17	1.000340	-1.167343	-0.763209	-0.336552	0.398625	2.867626
MonthlyRate	1470.0	3.021015e-17	1.000340	-1.717284	-0.880644	-0.010906	0.864101	1.782888
NumCompaniesWorked	1470.0	2.693197e+00	2.498009	0.000000	1.000000	2.000000	4.000000	9.000000
PercentSalaryHike	1470.0	1.520952e+01	3.659938	11.000000	12.000000	14.000000	18.000000	25.000000
RelationshipSatisfaction	1470.0	1.712245e+00	1.081209	0.000000	1.000000	2.000000	3.000000	3.000000
StockOptionLevel	1470.0	7.938776e-01	0.852077	0.000000	0.000000	1.000000	1.000000	3.000000
TotalWorkingYears	1470.0	1.127959e+01	7.780782	0.000000	6.000000	10.000000	15.000000	40.000000
TrainingTimesLastYear	1470.0	2.799320e+00	1.289271	0.000000	2.000000	3.000000	3.000000	6.000000
WorkLifeBalance	1470.0	1.761224e+00	0.706476	0.000000	1.000000	2.000000	2.000000	3.000000
YearsAtCompany	1470.0	7.008163e+00	6.126525	0.000000	3.000000	5.000000	9.000000	40.000000
YearsInCurrentRole	1470.0	4.229252e+00	3.623137	0.000000	2.000000	3.000000	7.000000	18.000000
YearsSinceLastPromotion	1470.0	2.187755e+00	3.222430	0.000000	0.000000	1.000000	3.000000	15.000000
YearsWithCurrManager	1470.0	4.123129e+00	3.568136	0.000000	2.000000	3.000000	7.000000	17.000000

	Age	Attrition	BusinessTravel	DailyRate	Department	DistanceFromHome	Education	EducationField	EmployeeCount	EmployeeNumber	...	RelationshipSatisfaction	StandardHours	StockOptionLevel	TotalWorkingYears	TrainingTimesLastYear	WorkLifeBalance	YearsAtCompany	YearsInCurrentRole	YearsSinceLastPromotion	YearsWithCurrManager
0	41	Yes	Travel_Rarely	1102	Sales	1	2	Life Sciences	1	1	...	1	80	0	8	0	1	6	4	0	5
1	49	No	Travel_Frequently	279	Research & Development	8	1	Life Sciences	1	2	...	4	80	1	10	3	3	10	7	1	7
2	37	Yes	Travel_Rarely	1373	Research & Development	2	2	Other	1	4	...	2	80	0	7	3	3	0	0	0	0
3	33	No	Travel_Frequently	1392	Research & Development	3	4	Life Sciences	1	5	...	3	80	0	8	3	3	8	7	3	0
4	27	No	Travel_Rarely	591	Research & Development	2	1	Medical	1	7	...	4	80	1	6	3	3	2	2	2	2

	Age	Attrition	DailyRate	DistanceFromHome	Education	EmployeeCount	EmployeeNumber	EnvironmentSatisfaction	HourlyRate	JobInvolvement	...	JobRole_Research Director	JobRole_Research Scientist	JobRole_Sales Executive	JobRole_Sales Representative	MaritalStatus_Married	MaritalStatus_Single	OverTime_Yes	StockOptionLevel_1	StockOptionLevel_2	StockOptionLevel_3
0	41	Yes	1102	1	2	1	1	1	94	2	...	False	False	True	False	False	True	True	False	False	False
1	49	No	279	8	1	1	2	2	61	1	...	False	True	False	False	True	False	False	True	False	False
2	37	Yes	1373	2	2	1	4	3	92	1	...	False	False	False	False	False	True	True	False	False	False
3	33	No	1392	3	4	1	5	3	56	2	...	False	True	False	False	True	False	True	False	False	False
4	27	No	591	2	1	1	7	0	40	2	...	False	False	False	False	True	False	False	True	False	False

IBM HR Analytics Employee Attrition & Performance¶

PREPARED BY : Raj Purohith Arjun¶

UIN : 535005964¶

STAT 650¶

1. Overview¶

Research Background¶

Objective¶

Research Questions¶

Statistical Hypothesis¶

2. Dataset Description¶

Dataset Characteristics:¶

Variables Description¶

Dependent Variable (Target):¶

Independent Variables:¶

Irrelevant Variables:¶

Information about Dataset¶

3. Data Pre-Processing¶

3.1 Handling Missing Values and Duplicates¶

Explanation:¶

3.2 Encoding Categorical Variables¶

3.2.1 Label Encoding for Ordinal Variables:¶

3.2.2 One-Hot Encoding for Nominal Variables:¶

3.3 Scaling and Normalization¶

Dropping Irrevelant Columns¶

4. Exploratory Data Analysis (EDA)¶

4.1 Univariate Analysis¶

4.1.1 Summary Statistics¶

4.1.1 Summary Statistics¶

4.1.2 Visualization¶

Histogram¶

Box Plot¶

Density Plot¶

Key Insights and Findings¶

4.2 Bivariate Analysis: Analyzing Relationships and Dependencies¶

Corelation Matrix¶

Scatter Plot¶

Pair Plot¶

4.3 Multvariate Analysis¶

1. Heat Map¶

2. Pair Plot¶

3. Principal Component Analysis¶

5 Regression Analysis¶

5.1 Simple Linear Regression¶

Explanation¶

5.2 Multi Linear Regression¶

Explanation¶

5.3 Polynomial Regression(degree = 3)¶

Explanation and Compararision with linear regression¶

5.4 Logistic Regression¶

Explanation¶

5.5 Regularization Techniques¶

5.5.1 Lasso Regression , Ridge Regression and Elastic Net Regression¶

Explanation¶

5.5.2 Advanced Regression Techniques¶

5.5.2.1. Quantile Regression¶

Key Use Case:¶

5.5.2.2. Poisson Regression¶

Key Use Case:¶

5.5.2.3. Negative Binomial Regression¶

Key Use Case:¶

5.5.2.4. Zero-Inflated and Hurdle Regression¶

Key Use Case:¶

5.5.2.5. Cox Regression (Cox Proportional Hazards Model)¶

Key Use Case:¶

5.5.2.6. Partial Least Squares Regression (PLSR)¶

Key Use Case:¶

5.5.2.7. Principal Component Regression (PCR)*¶

Key Use Case:¶

Conclusion on Advanced Regression Techniques¶

6. Selected Model Evaluation and Validation¶

K Fold¶

Model Evaluation and Comparison of All Models¶

Evaluation Metrics for All Models¶

Regression Models¶

Classification Model¶

Comparison of Model Performance¶

Regression Models¶

Classification Model¶

Overall Comparison¶

7.Results and Interpretation¶